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ABSTRACT 



This paper discusses the results of a study that 
investigated the validity. and reliability of the Ecology Rating Scale (ERS) . 
The ERS is a brief, multi -dimensional level-of- functioning instrument that 
can be rated by parents or clinicians. The ERS is comprised of seven domains 
of youth functioning: family, school, emotional, legal/ justice, recreational, 
health, and social. For each domain, respondents are asked to assess the 
degree to which the child's life is influenced in the particular domain by 
problems associated with his/her emotional/behavioral difficulties. Each 
domain is rated on a 5-point scale of severity. Behavioral anchors are 
written on the scale itself for moderate (3) and severe (5) levels. The 
validity study involved 74 parents of children (ages 4-18) who were current 
clients at one of six community mental health centers in Washington state. 
Results found interrater reliability was strongest for social, recreation, 
and legal subscales. Other scales had weak interrater reliability. With the 
exception of the legal subscale of the ERS, all ERS scales provided some 
evidence of concurrent validity. (Contains 11 references.) (CR) 
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Introduction 

Child psychiatric epidemiology is a rapidly developing field 
(e.g., Brandenburg, Friedman, & Silver, 1990; Costello, 
1989; Koot & Verhulst, 1992), and with it, the need for 
adequate assessment tools has become apparent. Mental 
health concerns in children can be assessed in a variety of 
ways including: (a) psychiatric diagnosis using the 
DSM-IV; (b) level-of-functioning (LOF); (c) diagnosis plus 
LOF; and (d) exposure to risk factors for mental illness. 
Most commonly, diagnosis plus indication of LOF is used 
(Costello, Bums, Angold, & Leaf, 1993). However, there is 
a lack of adequate LOF instrumentation for youth. 



One instrument often used to measure LOF is the Children's 
Global Assessment Scale (CGAS; Schaffer et al., 1983). 

This single-item 0-100 scale was designed to be parallel in 
structure to the Global Assessment of Functioning scale for 
adults (GAF; Endicott, Spitzer, Fleiss, & Cohen, 1976) 
which is rated as axis V of a DSM diagnosis. Both the 
CGAS and GAF are intended to be rated by trained 
clinicians. A weakness of uni-dimensional scales is that they 
provide little information about the nature and complexity 
of children's mental health problems. Some youth may have 
rather prominent emotional/behavior problems in one 
setting (e.g., home/family, school) but not others. A single 
rating does not capture such subtleties. Similarly, certain 
domains are not recognized at all by such a scale (e.g., legal, 
health). Further, only multi-dimensional scales provide 
adequate information on which to describe clinically similar 
groups of youth that can provide the basis for designing 
services and service packages (e.g., Smukler, Sherman, 
Srebnik, & Uehara, in press). 

The Child and Adolescent Functioning Assessment Scale 
(CAFAS; Hodges, Bickman, & Kurtz, 1991) addresses 
some of the weaknesses of uni-dimensional LOF scales. The 
measure assesses five domains of child functioning: role 
performance, thinking, behavior, moods/emotions, and 
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substance use. Caregiver resoitrces are also rated. Each 
domain is rated on a 4-point severity scale with behavioral 
anchors for each rating. This scale is appealing due to its 
rich behavioral descriptions for each functioning domain, 
and it has been used in large-scale mental health services 
projects (e.g., Fort Bragg). However, extensive training of 
interviewers is needed to obtain adequate inter-rater 
reliability. Furthermore, the interview itself, on which 
ratings are based, is lengthy. 

Given the limitations of available LOF instruments for 
children and adolescents, development of a brief 
multi-dimensional LOF instrument, that requires little or no 
training to be rated by clinicians as well as parents, would 
be a valuable contribution to the field. This paper presents 
data describing the development and preliminary data on 
such an instrument, the "Ecology Rating Scale." 

Methods 

Participants 

The study collected data from 74 parents of children (age 
4-18), who were current clients at one of six community 
mental health centers in two large geographic regions of 
Washington state. The regions encompassed rural, suburban 
and small urban centers. Regions were selected as part of a 
larger systems' evaluation assessing the impact of replacing 
mental health process regulations with clinical outcomes as 
a method of accountability. 

The sample was 71% male and 13% ethnic minority. 
Ninety-three percent were Medicaid-eligible. Diagnoses 
were primarily adjustment disorder (43%), attention deficit 
(23%), oppositional defiant disorder (10%), and 
depression/dysthymia (10%). The sample was representative 
of the total served child populations of the selected regions 
in terms of gender, age, diagnostic mix, and income type. 
Participants who completed baseline surveys were contacted 
to complete assessment materials again six months later. 

Case managers of participating child clients were also asked 
to complete a brief assessment of client functioning and 
symptoms. Of the 74 participating child clients, 41 case 
manager assessments were completed. 

Instruments 

Ecology Rating Scale. The Ecology Rating Scale (ERS) 
was designed by quality assurance and children's mental 
health service staff at a community mental health center 
with input from families of child clients. The ERS is a brief, 
multi-dimensional level-of-functioning instrument that can 
be rated by parents or clinicians. The ERS is comprised of 
seven domains of youth functioning: family, school, 
emotional, legal/justice, recreational, health, and social (see 
Figure U . For each domain, respondents are asked to 
"assess the degree to which the child's life is influenced," in 
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"assess the degree to which the child's life is influenced," in 
the particular domain, by problems associated with his/her 
emotional/behavioral difficulties. Each domain is rated on a 
5-point scale of severity from no problems to severe 
problems that represents that degree to which the domain is 
influenced by the child's emotional/behavioral problems. 
Behavioral anchors are written on the scale itself for 
moderate (3) and severe (5) levels. 

Conner’s Parent Rating Scale. The Conner's Parent Rating 
Scale (Conners, 1970), is comprised of 48 children's 
symptoms and behavior problems, each rated on a 4-point 
severity scale. The instrument is appropriate for use with 
children age 3 to 17 and when age-by-gender norms are 
available. The instrument yields five factors: conduct 
problems, learning problems, psychosomatic symptoms, 
impulsive-hyperactive symptoms, and anxiety symptoms. 
Adequate reliability and validity have been demonstrated 
with this instrument (Conners, 1985). 

Children’s Global Assessment Scale. For participating 
child clients, case managers completed a brief functioning 
and symptom assessment that included the Ecology Rating 
Scale and the Children's Global Assessment Scale (CGAS; 
Shaffer et al, 1983) described above. Studies have 
demonstrated adequate inter-rater reliability as well as 
concurrent and discriminant validity for the CGAS (Bird, 
Canino, Rubio-Stipec, & Ribera, 1987; Shaffer, et al., 1983; 
Steinhausen, 1987). 

Information Systems Data. Data gathered from 
participating agency's information systems provided 
information on client's socio-demographic characteristics, 
residential situation, work/school involvement, and service 
utilization. 

Results 

The first step in analysis of the adequacy of the ERS was to 
examine the distribution of scale scores. Scale scores were 
normally distributed with the exception of scores for legal 
and health scales, domains for which few children 
demonstrated difficulties. Reliability and validity were then 
examined and findings are described below. 

Reliability 

The reliability of the ERS was examined with respect to 
both internal consistency and inter-rater reliability. While 
the ERS is composed of seven domains, each believed to 
contribute unique information, we did not expect domain 
ratings to be completely orthogonal. Rather, we expected 
moderate to high internal consistency for the scale as a 
whole. Results supported this expectation with alpha 
coefficients of .70 for parent ratings (N = 65) and .69 for 
clinician ratings (N = 41). 

Inter-rater reliability was tested with ERS scores from 27 
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Inter-rater reliability was tested with ERS scores from 27 
clients rated by two clinicians who reportedly knew the 
clients well. Results suggested that reliability was 
potentially adequate for family, legal, recreation, and social 
domains, with intra-class correlations of .50, .65, .71., and 
.83 respectively. Intra-class correlations for the remaining 
scales were less than adequate and are as follows: school 
(.44), emotions (.37), and health domains (.38). If used as an 
overall index of functioning, however, the total ERS score 
demonstrated strong inter-rater reliability with an intra-class 
correlation of .75. 

Validity 

Concurrent validity was first examined as the correlation 
between clinician-rated ERS scores and the CGAS (N = 41). 
Although school and emotion domains demonstrated 
relatively weak inter-rater reliability, they were among the 
scales with stronger concurrent validity with the CGAS, 
with correlations of .62 and .54., respectively, both 
significant at the p < .001 level. The recreation (r = .49) and 
social scales (r = .45) were significantly correlated to the 
CGAS at the p < .01 level. Family, legal, and health 
domains were moderately, though non-significantly, 
correlated with the CGAS with correlations ranging from r 
= .27 to .41 . The total ERS score was strongly and 
significantly correlated with the CGAS (r = .72, p < .001). 

Correlations between parent-rated ERS scores and subscale 
scores of the Conner's Parent Rating Scale were conducted 
as a second means of obtaining concurrent validity 
information. Predicted relationships are shown in Table I . 

Consistent with expectations, parents' perceptions of 
problems in the "family" and "emotions" domains were 
associated with externalizing behaviors such as the 
"impulsive-hyperactivity" and "conduct" subscales. Also, as 
predicted, problems in the "school" domain were associated 
with learning problems. Health problems were related to 
anxiety, consistent with expectations regarding the 
relationship of somatization with anxiety. Social and 
recreation problems were associated with conduct 
difficulties. This relationship is reasonable considering that 
youth with conduct issues will likely have difficulty getting 
along with others socially and in recreational settings (as 
well as within the family). Legal problems were also 
predicted to be associated with the conduct subscale; and 
the lack of association could be due to restriction of range 
on the legal domain scale. The ERS total score, as rated by 
parents, appears to be most highly related to externalizing 
behavior problems and learning problems, rather than 
internalizing problems. 

Treatment validity was tested through examination of 
whether ERS scores change over the course of six months of 
outpatient mental health treatment. The numbers of 
individuals who demonstrated change, and whether the 
sample change was significant, according to t-test 
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sample change was significant, according to t-test 
comparisons for matched samples is shown in Tables 2 and 
3. 

Table 2 shows parent ERS ratings, and Table 3 shows case 
manager ERS ratings. The number of individuals showing 
improvement relative to those demonstrating no change or 
declining functioning according to parents is greatest for the 
following domains: school, emotion, and legal. Case 
managers also reported many more individuals improving 
regarding the recreation domain. Overall sample change on 
the parent-rated ERS scales was significant, and in the 
expected direction, for two domains. Trends in the expected 
direction were found for the other ERS scales. Significant 
change was found on all but the "social" domain for case 
manager-rated ERS scales. 

Discussion 

This presentation described a brief, multi-dimensional, EOF 
instrument for youth, the Ecology Rating Scale (ERS). The 
ERS assesses functioning across 7 life domains: school, 
emotions, legal, health, recreation, social, and family. In 
contrast to extant measures, the ERS has the following 
unique combination of attributes: (a) assesses multiple 
domains of functioning, (b) requires little training, and (c) 
demonstrates promising psychometric properties. While we 
do not advocate use of any EOF tool without proper 
training, the psychometric characteristics of the ERS 
reported here were found without clinicians having any 
training. Ideally, clinicians using such instruments would 
periodically conduct formal or informal tests of inter-rater 
agreement in the course of discussing cases among clinical 
team members. 

Study findings regarding the ERS suggested some areas of 
internal consistency. Inter-rater reliability was strongest for 
social, recreation, and legal subscales. Other scales had 
weak inter-rater reliability, perhaps due to assessing 
behaviors related to non-clinical settings (e.g., school) or 
less recognizable internal states (e.g., emotions and health). 
Hodges (1991) also found weak inter-rater agreement for 
scales assessing internal states. On the other hand, more 
behavioral anchors combined with rater training may 
improve inter rater agreement of the ERS. 

Concurrent validity was demonstrated for ERS parent 
ratings with the Conner's behavior rating scale. Clinician 
ratings on ERS scales were also correlated with a 
uni-dimensional EOF measures, the Children's Global 
Assessment Scale. With the exception of the "legal" 
subscale of the ERS, all ERS scales provided some evidence 
of concurrent validity. Correlations with the legal subscale 
were likely affected by restriction of variance of responses 
on the subscale. Evidence of treatment validity (i.e., 
significant change in expected direction following 6 months 
of treatment) is also demonstrated for ERS scales. 
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Further development of the ERS is needed. In addition to 
developing more behavioral anchors for scale points, norms 
for the scale should be developed on a larger sample that 
includes higher rates of ethnic minority youth. 

Preliminary testing of the ERS demonstrates that it shows 
promise as a brief, practical tool for EOF assessment in 
youth receiving community mental health services. Uses for 
such a tool include clinical outcome assessment, 
performance measurement, continuous quality 
improvement, and services research. 
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